chore: stop unit test OOM from a spinning synchronizer loop - #390

What happened

FDv2DataSourceTest.recoveryResetsToFirstAvailableSynchronizer intermittently fails CI with OutOfMemoryError, reported against tearDown's executor.shutdownNow() and Arrays.copyOf. It is unrelated to whatever PR happens to be running; it is a pre-existing flake in this test.

Root cause

The test's synchronizer factories return the same MockQueuedSynchronizer instance on every build() call, while the real factories construct a new one. SourceManager.getNextAvailableSynchronizerAndSetActive() closes the synchronizer it is switching away from, and MockQueuedSynchronizer.next() on a closed instance returns an already-completed SHUTDOWN.

So once recovery has cycled through both slots, every instance is closed and the inner loop in runSynchronizers exits immediately, while the outer loop is round-robin (getNextAvailableSynchronizer wraps its index) and re-enters with no delay. Each pass logs Synchronizer '{}' is starting. into LogCaptureRule's LogCapture, which retains every message for the life of the test, so the live heap grows until the JVM is exhausted.

Instrumenting the factories shows the scale of the spin. In a single ~10 second run of this one test, each factory was invoked about 630,000 times, where the test intends roughly two:

PROBE executorQueue=0 firstCalls=629883 secondCalls=629883

And live heap after each full GC climbs monotonically, never recovering:

Pause Full 2047M->1808M(2048M)
Pause Full 2041M->1912M(2048M)
...
Pause Full 2038M->2026M(2048M)

The fix

Build a fresh synchronizer per factory call, as production does. A new instance is never pre-closed, so next() serves its queued results and then returns a future that only completes later, which restores the pacing the test intended: the cycle waits on the 1s fallback and 2s recovery timers instead of spinning. Live heap after full GC then stays flat at ~75M.

Why the stack trace points at shutdownNow()

That is where the OOM surfaces, not where the memory went. shutdownNow() calls drainQueue(), whose toArray needs a fresh allocation (Arrays.copyOf), and on an already-exhausted heap that is simply the allocation that happens to fail. As the probe above shows, the executor's queue is empty (executorQueue=0): during the spin every result is SHUTDOWN, so the fallback timer never arms, and the recovery timer is not built once the index has been reset to the prime synchronizer. The retained memory is the accumulated log capture, not scheduled tasks or pending futures.

Test plan

• ./gradlew test passes locally
• Verified the live heap after full GC stays flat (~75M) with the fix, instead of climbing to 2026M of 2048M
• Verified at the current 512m default test heap, so the fix does not depend on any heap change

Nothing shipped changes, so this is chore and should not cut a release.

Related

Raising the forked test JVM's heap is split out into its own PR as headroom. It does not fix this OOM (the retention fills any heap — it still OOMs at 2g), so this PR stands alone.

Follow-up worth considering (not in this PR)

The hot loop is reachable in principle outside tests: if every synchronizer reports SHUTDOWN immediately, runSynchronizers spins round-robin with no backoff, logging each pass. On a device that means CPU burn and log spam. Production factories build fresh synchronizers that block on I/O, so this is unlikely in practice, but a guard or backoff in that loop may be worth adding by whoever owns FDv2.